Automatic correction of skewing of digital images

ABSTRACT

A method and system. An initial angle of rotation is determined by sampling test angles of rotation of a captured digital image and analyzing resultant rotated images to determine a resultant rotated image with a highest number of substantially empty lines. The captured digital image is rotated by the initial angle of rotation to generate a first rotated image. A representative line of each line area of multiple text line areas of the first rotated image is generated. A slope of each representative line is calculated. An aggregated slope of the representative lines is determined. The aggregated slope is converted to a refining angle of rotation. The refining angle of rotation is validated by finding points of intersection of lines connecting opposing ends of adjacent representative lines. The first rotated image is rotated by the refining angle of rotation to result in a final rotated image.

This application is a Continuation application claiming priority to Ser.No. 14/878,547, filed Oct. 8, 2015, now U.S. Pat. No. 9,621,761, issuedApr. 11, 2017.

TECHNICAL FIELD

The present invention relates to automatic correction of digital images,and more specifically, to automatic correction of skewing of digitalimages.

BACKGROUND

Scanned documents can be skewed due to common challenges with the imagecapturing process. This applies to images of documents taken byscanners, digital cameras including mobile cameras, or other capturingapparatus.

SUMMARY

Embodiments of the present invention provide a computer-implementedmethod, and associated system and computer program product, forautomatic correction of skewing of digital images. A captured digitalimage for processing as an input image is provided. An initial angle ofrotation is determined by sampling a plurality of test angles ofrotation of the input image and analyzing resultant rotated images todetermine a resultant rotated image with a highest number ofsubstantially empty lines. The input image is rotated by the initialangle of rotation to generate a first rotated image. The first rotatedimage is processed to determine a refining angle of rotation, including:determining text line areas of the first rotated image; generating arepresentative line of each text line area; calculating a slope of eachrepresentative line; and determining an aggregated slope of all therepresentative lines in the first rotated image, wherein the aggregatedslope is converted to the refining angle of rotation; and rotating thefirst rotated image by the refining angle of rotation to result in afinal rotated image.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter regarded as the invention is particularly pointed outand distinctly claimed in the concluding portion of the specification.The invention, both as to organization and method of operation, togetherwith objects, features, and advantages thereof, may best be understoodby reference to the following detailed description when read with theaccompanying drawings.

FIG. 1 is a flow diagram of an example embodiment of the method of thepresent invention.

FIG. 2 is a flow diagram of further details of the method of FIG. 1.

FIG. 3 is a flow diagram of further details of the method of FIG. 1.

FIG. 4 is a flow diagram of further details of the method of FIG. 1.

FIG. 5 is a flow diagram of further details of the method of FIG. 1;

FIGS. 6A to 6F are schematic diagrams illustrating the method of thepresent invention as applied to a captured image, in accordance withembodiments of the present invention.

FIG. 7 is an example of captured image to which the method of thepresent invention has been applied, in accordance with embodiments ofthe present invention.

FIG. 8 is block diagram of an example embodiment of a system inaccordance with embodiments of the present invention.

FIG. 9 is a block diagram of an embodiment of a computer system in whichthe present invention may be implemented.

It will be appreciated that for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn to scale.For example, the dimensions of some of the elements may be exaggeratedrelative to other elements for clarity. Further, where consideredappropriate, reference numbers may be repeated among the figures toindicate corresponding or analogous features.

DETAILED DESCRIPTION

Skewing of an image not only makes reading a document more difficult,but it also affects the quality of Optical Character Recognition (OCR)that can be applied to the captured document.

Therefore, solving this skewing problem would result in improvedcapability for computers to process the content of scanned documents andallow digital systems to deal with data that is sourced from hardcopies,captured using cameras or any digital image capturing device.

The problem is not new and there are several solutions in place each ofwhich has its own limitations.

The use of Hough transforms is known in this field, however use of Houghtransforms is dependent on the quality of the input data. The transformis less efficient when dealing with noise. For example, edge detectionmay significantly suffer if there is a watermark, a stain, or the edgesare distorted. Furthermore, the transform when applied on its own doesnot provide a means of validating the accuracy of the transform. Inother words, the accuracy of a method that depends on Hough transform ishighly sensitive to the quality of the content of the text within thetext lines. Unless otherwise stated or implied, “pixel line”, “textline”, and “line” are synonymous in this description of the invention.

If the quality of scanned text and/or physical paper is poor, theaccuracy of the transform suffers.

The use of Fast Fourier transforms is also known in this field. Theirregular pattern of letters (and other scan or page deformations)results in considerable noise within the transform result which requiresfurther costly processing such as applying morphological operators.

Therefore, there is a need in the art to address the aforementionedproblems.

The described method and system of the present invention, in oneembodiment, finds an angle of rotation that makes a captured digitalimage look as much as possible like a page with text lines that areparallel or substantially parallel to each other. Text lines may besubstantially parallel, for example, if the text lines are handwrittenwithout guiding lines.

Referring to FIG. 1, a flow diagram 100 shows an example embodiment ofthe method of the present invention.

A captured digital image may be provided (step 101) for processing. Thedigital image may be captured from an original document by scanning, bya digital camera, or may be stored on some other electronic device. Thecaptured digital image may be moved (e.g., rotated) within a fixedrectangular coordinate system that is fixed in a background or “page” onwhich the digital image appears. The rectangular coordinate system has ahorizontal axis and a vertical axis which are perpendicular to eachother. The horizontal axis defines a horizontal direction and thevertical axis defines a vertical direction. If the captured digitalimage were not skewed, then the text lines of the digital image would beparallel to the horizontal direction. Unless otherwise stated orimplied, all angles in this description of the invention are relative toa reference direction in the rectangular coordinate system. Thereference direction may be any fixed direction (e.g., the horizontaldirection, the vertical direction, etc.) in the rectangular coordinatesystem.

Pre-processing (step 102) of the captured digital image may be carriedout to reduce noise in the image. The image may be modified for bettersubsequent processing by using methods to filter out part of the noise(e.g. color, contrast, font used, text language, stains, deformed edges,handwritten marks not written in lines, etc.). This pre-processingimproves the accuracy of deskewing.

An initial rotation angle is then calculated (step 103) as the initialrotation angle that results in the best number of line spacings afterpre-processing the input image to reduce the effect of noise. The bestnumber of line spacings is determined by the highest number ofsubstantially empty lines of pixels in a rotated image. Further detailsof step 103 are given with respect to FIG. 2. Step 103 may be arelatively low processing cost method to find an initial angle ofrotation before carrying out a more intense processing from the initialangle.

After rotating (step 104) the digital image at the best initial rotationangle, text line areas are detected (step 105) as areas of adjacentlines, each line with a sufficient or threshold number of non-emptypixels, wherein there is a text line area between two empty lines.Further details of step 105 are given with respect to FIG. 3.

An individual representative line for each text line area is determined(step 106) and a deskew angle is calculated (step 107) for eachrepresentative line by finding individual slopes of representativelines. A refined deskewing angle is calculated (step 108) based on theaggregated slopes of the representative lines of all the detected textline areas in the digital image. Further details of steps 106-108 aregiven with respect to FIG. 4.

The deskew rotation angle is calculated by taking into consideration theslope of each individual representative line even if the text lines arenot uniform or parallel to each other.

Lines may be detected by combining the three parameters together of linespacing, density of lines of points, and density of points comprising aline, for example. Threshold values may be used to further reduce theweight of noise mentioned above.

Thresholds may be calculated in a relative manner based on values thatare driven from the same image characteristics. For example, ifstatistically most of the lines' slopes are between 35 and 40 degrees,and one line is identified as having a slope of 70 degrees, a thresholdof 40 degrees as an upper limit and 35 degrees as a lower limit may beapplied with exclusion of any lines outside the thresholds. A similarapproach may be made to a line's height. For example, a minimum textline height may be based on the height of all detected text lines on theimage.

Optionally, a secondary validation may be carried out (step 109) using,for example, a substantially vertical line cutting the averaged centersof detected representative text lines to determine the accuracy of thedeskewing. This secondary validation is based on the aggregatedrelationship of each two consecutive text lines to assess the quality ofthe deskew rotation angle calculated above. Further details of step 109are given with respect to FIG. 5.

Referring to FIG. 2, a flow diagram 200 shows a further exampleimplementation detail of step 103 of FIG. 1 of calculating an initialrotation angle of a digital image. A best initial rotation angle isdetermined before the angle is fine-tuned with further analysis asdescribed in the subsequent flow diagrams.

A digital image is provided (step 201), which may be acquired byscanning, taken by a digital camera, or already stored on someelectronic device.

A range of angles to be selected is determined (step 202) as an inputparameter. For example, the range of angles can be from −90 degrees to+90 degrees. Although the algorithm is capable of going beyond thisrange, it is not mended as it may result in rotating the image upsidedown of its intended orientation.

For each incremented angle of the range, the following steps may becarried out. The incremented angle may be chosen so that the angle isnot too large as to miss the optimum angle, whilst being not too smallas to increase the resource consumption of processing.

The input image may be pre-processed (step 203) which is similar to thepre-processing carried out at step 102 of FIG. 1, but is more costly atthis stage. Therefore, the pre-processing may only be carried out whenthe method has more confidence in the initial image orientation angle.At this stage, the original image and carry out additionalpre-processing methods are analyzed.

The image may be converted (step 204) to binary black and white or grayscale to reduce the effect of colors and physical deformations thatmight have been on the originally provided digital image.

A blur filter may be applied to the image to reduce the effect ofindividual parts of a text line on the overall text line area calculatedslope. Application of the blur filter also reduces the effect of noisethat might be scattered within or outside text lines across the page.Since the method is dealing with the concentration of points in the formof pixels, the blur filter filters out scattered individual points thatare not typically part of text lines (for example, a black dot that wasprinted out by a printer, or a piece of small dust on the originalhardcopy). The blur filter reduces the effect of such noise.

A bitmap may be created of the image where the number of bits in anypixel below a set pixel content threshold is considered white;otherwise, the pixel is considered black. The pixel content threshold isexpressed as a threshold number of bits. The pixel content threshold maybe one of the input parameters to the algorithm. This binarization ofwhite/black reduces the noise of weak spots regardless of their size onthe page and at the same time reduces the number of points to beprocessed with the algorithm to save computing power.

Using the image bitmap, a weight is given (step 205) for each pixel linebased on the number of black pixels on that pixel line.

Using the weights calculated, empty pixel lines and the non-empty pixellines are determined. That is, any pixel line whose weight is less thana set line content threshold (expressed as a threshold number of blackpixels) is considered empty; otherwise, the pixel line is considerednon-empty. This weight of a pixel line relative is a measure of how weakthe pixel line is. The set line content threshold may be one of theinput parameters to the algorithm. The number of detected pixel linesconsidered empty is determined (step 206).

It is then determined (step 207) if all the angles in the selected rangehave been processed. If all the angles in the selected range have notbeen processed, then the method loops to step 202 and a next angle hassteps 203 to 206 applied. Once all angles have been processed, themethod continues at step 208.

An initial rotation angle is set (step 208) to the value of the anglethat resulted in the largest number of empty lines, which means that thetext lines on the page are likely to be ordered in a parallel orsemi-parallel manner at this angle. The method then proceeds (step 209)to the method of the flow diagram 300 of FIG. 3.

Referring to FIG. 3, a flow diagram 300 illustrates further exampleimplementation detail of step 105 of FIG. 1 of detecting text line areasas areas of adjacent lines, each line having a threshold number ofnon-empty pixels, wherein the text line area is between two empty lines.

The initially rotated pre-processed image and a list of empty/non-emptypixel lines are provided (step 301).

For each pixel line of the image (step 302), the following method iscarried out.

It is determined (step 303) if the pixel line is part of a new text linearea. If the pixel line is not part of a new text line area, it isdetermined (step 304) if there are remaining detected pixel lines from aprevious cycle of the method. If so, the remaining detected pixel linesare added (step 305) as an area to a list.

If it is determined in step 303 that the pixel line is part of a newtext line area, or after steps 304 and 305 have been carried out, it isthen determined (step 306) if the weight of the pixel line is below apixel line threshold. The pixel line weight of the pixel line is thetotal number of non-empty pixels in the pixel line. The pixel linethreshold is expressed as a threshold number of non-empty pixels. If thepixel line weight is below the pixel line threshold, the pixel line ismarked (step 307) as a new area in a next cycle. If the pixel lineweight is at or above the pixel line threshold, the pixel line is added(step 308) to the current area.

It may then be determined (step 309) if all lines have been processed.If all lines have not been processed, the method loops to step 303 toprocess a next line.

For each pixel line area of the image, the pixel line is added to thecurrent text line area if the weight of the pixel line is at or abovethe pixel line threshold. This pixel line threshold can be set as aninput parameter or can be calculated based on the relative weights ofother pixel lines on the page (e.g. the 60 percentile of all pixellines). A new text line area is triggered when an empty line isencountered. The weight of the pixel line measures how strong the pixelline is and is measured according to what is similar to the other pixellines. Detected text line areas that contain less non-empty pixel linesthan a text area line threshold are removed. The text area linethreshold is expressed as a threshold number of non-empty pixel lines.This text area line threshold can be set as an input parameter or can becalculated relative to the rest of text area lines on the page (e.g. the50 percentile of all text area lines).

If all the lines have been processed, areas are removed (step 310) fromthe list with too few pixel lines. The method then proceeds (step 311)to the method of the flow diagram 400 of FIG. 4.

Referring to FIG. 4, a flow diagram 400 shows further exampleimplementation detail of steps 106, 107 and 108 of FIG. 1 of findingrepresentative lines of each text line area, determining a slope of eachrepresentative lines, and determining an aggregated slope of all therepresentative lines in a digital image.

The list of text line areas is provided (step 401). For each area in thelist, the following is carried out (step 402).

A line that represents the pixels of the area is calculated and stored(step 403). There are several calculation options that can be used. Inone embodiment, linear regression is used. In other embodiments, anygeometrical or statistical method may be used to obtain therepresentative line. Examples include calculating the mean point of asample of two vertical lines within a text area, or the median oraverage of several such sample points. The representative linerepresents the slope of the text area, taking into consideration thecharacters within the line may be above and/or below the representativeline. The representative line may be called the middle line forsimplicity.

The slope of the representative line and, optionally, the starting andending points of the representative line, is calculated and stored (step404). The starting point is where x=0 and the ending point is at x=imagewidth. The starting and ending points may be used for the optionalvertical verification at the end of the method.

It is determined (step 405) if all the text line areas of an image havebeen processed. If all the text line areas of an image have not beenprocessed, the method loops to step 402 to process the next text linearea.

If all the text line areas of an image have been processed, the methodmay calculate (step 406) the page deskew angle based on the slopes ofthe representative lines of the text line areas calculated above. Thereare several calculation options that can be used. In one embodiment, themedian is used to reduce the effect of the noise of extreme values(lines that are extremely skewed compared to the rest of the text linesdetected). The method then optionally proceeds (step 407) to the methodof the flow diagram 500 of FIG. 5.

Referring to FIG. 5, a flow diagram 500 shows further exampleimplementation detail of step 109 of FIG. 1 of running a secondaryvalidation of the angle using a vertical line determination.

The initially rotated image is provided (step 501). Starting at thesecond text line area and until the end of the text line areas, thefollowing steps are carried out (step 502).

The starting point and the ending point of the representative line ofthe current text line are determined (step 503). A straight line isconnected (step 504) between the starting point of the currentrepresentative line and an ending point of the previous representativeline and vice versa. The point of intersection of the preceding twolines is calculated (step 505).

It is then determined (step 506) if all the areas have been processed.If not, the method loops to step 502 to process the next area.

In response to all text line areas having been processed, a line, calleda “vertical line”, that closely represents the points of intersectionmentioned above is determined (step 507). There are several calculationoptions that can be used. In one embodiment, linear regression is used.The, the “vertical line” is a linear fit to the points of intersectionand is determined by the points of intersection. Thus the vertical lineis not required to be oriented in the vertical direction. However, thevertical line is oriented more closely to the vertical direction as theamount of skew of the digital image decreases.

The top and bottom points of the vertical line may be determined (step508) and used to calculate (step 509) the deviation of each of the twopoints and the center horizontal point of the image. The method may thenend (step 510). The top and bottom points of the vertical line are atthe intersection of the vertical line with the top and bottomrepresentative lines, respectively, in the vertical direction.

On a perfectly deskewed image, the vertical line is oriented parallel tothe vertical direction defined by the rectangular coordinate system. Fora skewed image, this vertical line will also be skewed, relative to thevertical direction, with the same angle relative to the verticaldirection as the angle that the skewed image makes with a perfectlydeskewed image. The maximum amount of horizontal deviation in thevertical line is at the most top (y=0) and the most bottom (y=imageheight) of that vertical line. If this horizontal deviation is above acertain threshold length that may be predefined as an input parameter tothe algorithm, the quality of the initial rotation will be consideredlow. The horizontal deviation of this vertical line from the verticaldirection may be optionally used in conduction with the deskew anglecalculated earlier.

Referring to FIGS. 6A to 6E, schematic diagrams illustrate the abovedescribed method. FIG. 6A shows a skewed input digital image 600.

FIGS. 6B, 6C and 6D show the input digital image 600 having the methodof FIG. 2 applied to determine an initial angle of rotation. The digitalimage 600 is shown in FIGS. 6B, 6C and 6D for three respective trialangles of rotation and for each trial angle, the number of empty areasis determined.

In FIG. 6B, only one empty area 601 is determined. In FIG. 6C, fourempty areas are determined 611, 612, 613, 614. In FIG. 6D, four emptyareas are also determined 621, 622, 623, 624 but they have a largernumber of empty lines within them (the empty areas are wider) andtherefore the initial angle of rotation of FIG. 6D is selected.

Referring to FIG. 6E, the digital image 600 with outcome of the methodsof FIGS. 3 and 4 is shown. The digital image 600 is shown with fiverepresentative lines 631, 632, 633, 634, 635 determined, one for each ofthe non-empty areas of FIG. 6D. An aggregated slope 640 is also shown asa median of the slopes of the five representative lines 631, 632, 633,634, 635. In one embodiment, the median slope is the slope of line 633since line 633 is a median line between lines 631-632 and 634-635. Inone embodiment, the median slope is the arithmetic average of the slopesof lines 632 and 633. In one embodiment, the median slope is a weightedaverage of the slopes of lines 631-632 (weight=2) and lines 633-635(weight=3). In one embodiment, the median slope is a randomly selectedslope between the slopes of lines 632 and 633.

Referring to FIG. 6F, the digital image 600 with the outcome of theoptional method of FIG. 5 is shown. Points of intersection 651, 652,653, 654 of lines connecting opposing ends of adjacent representativelines 631, 632, 633, 634, 635 are determined. A vertical line 660 isthen determined from the points of intersection 651, 652, 653, 654 asdiscussed supra (e.g., by linear regression). An angle of deviationbetween the vertical line 660 and a line perpendicular to the aggregatedslope 640 provides an indication of the accuracy of the aggregated slope640.

FIG. 7 illustrates an example showing an initial digital image 711 aprocessing of the image 712 and a resultant image 713 with the deskewingangle applied.

The described method advantageously reduces the effect of noise over theseries of method steps which allows for better ability to processscanned documents of low quality. The low quality may be for manyreasons, for example, due to an inefficient scanning process or due tothe poor state of the original scanned document.

The described method advantageously uses cost effective method whiletrying to reach the best starting rotation angle, whereas it uses a morecostly processing to fine tune the initial rotation angle based on thecontent of detected text lines.

The criterion for detecting the initial rotation angle is cheaper incalculation cost as it relies on the number of empty lines (linespacing) as opposed to the content of the lines detected. The methoddoes not process text line content while trying different rotationangles.

The described method includes an optional secondary validation based onanother characteristic of deviation of the vertical line from thevertical axis of the same content detected. This characteristic isdifferent from the one used to calculate the deskew angle which is thecollective slopes of text lines. The secondary validation aggregates thevalue of the characteristic over all detected text lines as opposed toindividual text lines which can lead to inaccuracy depending on thequality of text line detection.

The described method has no dependency on the language or the font ofthe text in the scanned document, the quality of the text or even havingparallel text lines. The described method also processes both machineprinted scanned documents or handwritten scanned documents where textlines do not have to be parallel or be skewed at the same angle.

The described method also covers a high rotation range of 180 degrees,from −90 to +90 degrees.

Referring to FIG. 8, a block diagram shows an example embodiment of adigital image processing system 800 implementing the described method.

The digital image processing system 800 may include at least one atleast one processor 801, a hardware module, or a circuit for executingthe functions of the described components which may be software unitsexecuting on the at least one processor.

Multiple processors running parallel processing threads may be provided,enabling parallel processing of some or all of the functions of thecomponents. Memory 802 may be configured to provide computerinstructions 803 to the at least one processor 801 to carry out thefunctionality of the components.

The digital image processing system 800 may include a digital imageinput component 810 for receiving a digital image from a scanner, cameraor from an existing memory file, and optionally storing the digitalimage in a storage medium or storage device 811 for processing.

A user configuration component 813 may be provided to enable a user oroperator of the system to include configuration parameters forcontrolling the digital image processing, including providing thresholdlevels for the processing. A pre-processing component 812 may carry outpre-processing of the digital image to reduce noise in the image.

An initial rotation component 820 may determine an initial angle ofrotation using the method of FIG. 2. This may include additionalcomponents of a pixel map component 821, a pixel line weightingcomponent 822, and an empty line determining component 823.

A refining rotation component 860 may be provided for processing thefirst rotated image to determine a refining angle of rotation by furtherprocessing. The refining rotation component 860 may carry out the methodof FIGS. 3 and 4. A text line area component 830 may determine text lineareas. A representative line component 831 may find individual bestlines for each text line area, and a representative line slope component832 may find the slope of each representative line. An aggregate slopecomponent 833 may find the aggregated slope of the representative lines.

A validation component 840 may carry out the method of FIG. 5 ofvalidating the aggregated slope. A vertical line component 841 mayconstruct a substantially vertical line and a comparison component 842may compare this to the aggregated slope and may output a validationresult.

An output component 850 may rotate the digital image by the determinedbest angle.

Referring to FIG. 9, an exemplary system for implementing aspects of theinvention includes a data processing system 900 suitable for storingand/or executing program code including at least one processor 901coupled directly or indirectly to memory elements through a bus system903. The memory elements may include local memory employed during actualexecution of the program code, bulk storage, and cache memories whichprovide temporary storage of at least some program code in order toreduce the number of times code must be retrieved from bulk storageduring execution.

The memory elements may include system memory 902 in the form of readonly memory (RUM) 904 and random access memory (RAM) 905. A basicinput/output system (BIOS) 906 may be stored in ROM 904. Software 907may be stored in RAM 905 including system software 908 such as operatingsystem software 909. Software applications 910 including one or moreapplications for implementing the method discussed with respect to FIG.1 may also be stored in RAM 905, elsewhere in system memory 902, inprimary storage means 911, and/or secondary storage means 912.

The system 900 may also include primary storage means 911 such as amagnetic hard disk drive and secondary storage means 912 such as amagnetic disc drive and an optical disc drive. The drives and theirassociated computer-readable media provide non-volatile storage ofcomputer-executable instructions, data structures, program modules andother data for the system 900. Software applications may be stored onthe primary and secondary storage means 911, 912 as well as the systemmemory 902.

The computing system 900 may operate in a networked environment usinglogical connections to one or more remote computers via a networkadapter 916.

Input/output devices 913 may be coupled to the system either directly orthrough intervening I/O controllers. A user may enter commands andinformation into the system 900 through input devices such as akeyboard, pointing device, or other input devices (for example,microphone, joy stick, game pad, satellite dish, scanner, or the like).Output devices may include speakers, printers, etc. A display device 914is also connected to system bus 903 via an interface, such as videoadapter 915.

The present invention may be a system a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD) amemory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

A computer program product of the present invention comprises a computerreadable hardware storage device having computer readable program codestored therein, said program code containing instructions executable bya processor of a computer system to implement the methods of the presentinvention.

A computer system of the present invention comprises a processor, amemory, and a computer readable hardware storage device, said storagedevice containing program code executable by the processor via thememory to implement the methods of the present invention.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

Improvements and modifications can be made to the foregoing withoutdeparting from the scope of the present invention.

What is claimed is:
 1. A method, said method comprising: determining aninitial angle of rotation by sampling a plurality of test angles ofrotation of a captured digital image and analyzing resultant rotatedimages to determine a resultant rotated image with a highest number ofsubstantially empty lines, wherein said determining the initial angle ofrotation does not process content of text lines resulting in lowprocessing costs; rotating the captured digital image by the initialangle of rotation to generate a first rotated image; generating arepresentative line of each line area of multiple text line areas of thefirst rotated image; calculating a slope of each representative line;determining an aggregated slope of all of the representative lines inthe first rotated image, wherein the aggregated slope is converted to arefining angle of rotation; validating the refining angle of rotation byfinding points of intersection of lines connecting opposing ends ofadjacent representative lines, and estimating a substantially verticalline through the point of intersection, wherein the difference betweenthe substantially vertical line and a line perpendicular to theaggregated slope indicates the accuracy of the refining angle ofrotation; and rotating the first rotated image by the refining angle ofrotation to result in a final rotated image.
 2. The method of claim 1,wherein said determining the initial angle of rotation, includes:sampling a plurality of test angles at regular angle intervals ofrotation.
 3. The method of claim 1, wherein said determining the initialangle of rotation, includes: sampling a plurality of test angles in arange of −90 degrees to +90 degrees rotation.
 4. The method of claim 1,wherein said determining the initial angle of rotation, includes foreach resultant rotated image of a sample test angle: converting theresultant rotated image into a pixel map with each pixel in the pixelmap being determined to be empty or non-empty by applying a pixelcontent threshold; weighting each line of pixels in the pixel mapaccording to a number of non-empty pixels in said each line; determiningan empty line in the pixel map as a line of pixels in the pixel map witha weight less than a line content threshold; and counting a number ofempty lines in the resultant rotated image.
 5. The method of claim 1,said method further comprising: determining the text line areas of thefirst rotated image by forming areas of adjacent pixel lines, whereinthe adjacent pixel lines are each above a threshold weight of non-emptypixels, and wherein the determined text line areas are separated by atleast one pixel line below the threshold weight of non-empty pixels. 6.The method of claim 1, wherein said generating the representative lineof each text line area uses a geometrical or statistical method ofaveraging a text line area.
 7. The method of claim 1, said methodfurther comprising: pre-processing the captured digital image to reducenoise by filtering extraneous detail from the captured digital image. 8.A computer system, comprising a processor, a memory coupled to theprocessor, and a computer readable storage device coupled to theprocessor, said storage device containing program code executable by theprocessor via the memory to implement a method, said method comprising:determining an initial angle of rotation by sampling a plurality of testangles of rotation of a captured digital image and analyzing resultantrotated images to determine a resultant rotated image with a highestnumber of substantially empty lines, wherein said determining theinitial angle of rotation does not process content of text linesresulting in low processing costs; rotating the captured digital imageby the initial angle of rotation to generate a first rotated image;generating a representative line of each line area of multiple text lineareas of the first rotated image; calculating a slope of eachrepresentative line; determining an aggregated slope of all of therepresentative lines in the first rotated image, wherein the aggregatedslope is converted to a refining angle of rotation; validating therefining angle of rotation by finding points of intersection of linesconnecting opposing ends of adjacent representative lines, andestimating a substantially vertical line through the point ofintersection, wherein the difference between the substantially verticalline and a line perpendicular to the aggregated slope indicates theaccuracy of the refining angle of rotation; and rotating the firstrotated image by the refining angle of rotation to result in a finalrotated image.
 9. The computer system of claim 8, wherein saiddetermining the initial angle of rotation, includes: sampling aplurality of test angles at regular angle intervals of rotation.
 10. Thecomputer system of claim 8, wherein said determining the initial angleof rotation, includes: sampling a plurality of test angles in a range of−90 degrees to +90 degrees rotation.
 11. The computer system of claim 8,wherein said determining the initial angle of rotation, includes foreach resultant rotated image of a sample test angle: converting theresultant rotated image into a pixel map with each pixel in the pixelmap being determined to be empty or non-empty by applying a pixelcontent threshold; weighting each line of pixels in the pixel mapaccording to a number of non-empty pixels in said each line; determiningan empty line in the pixel map as a line of pixels in the pixel map witha weight less than a line content threshold; and counting a number ofempty lines in the resultant rotated image.
 12. The computer system ofclaim 8, said method further comprising: determining the text line areasof the first rotated image by forming areas of adjacent pixel lines,wherein the adjacent pixel lines are each above a threshold weight ofnon-empty pixels, and wherein the determined text line areas areseparated by at least one pixel line below the threshold weight ofnon-empty pixels.
 13. A computer program product, comprising a computerreadable hardware storage device having computer readable program codestored therein, said program code containing instructions executable bya processor of a computer system to implement a method, said methodcomprising: determining an initial angle of rotation by sampling aplurality of test angles of rotation of a captured digital image andanalyzing resultant rotated images to determine a resultant rotatedimage with a highest number of substantially empty lines, wherein saiddetermining the initial angle of rotation does not process content oftext lines resulting in low processing costs; rotating the captureddigital image by the initial angle of rotation to generate a firstrotated image; generating a representative line of each line area ofmultiple text line areas of the first rotated image; calculating a slopeof each representative line; determining an aggregated slope of all ofthe representative lines in the first rotated image, wherein theaggregated slope is converted to a refining angle of rotation;validating the refining angle of rotation by finding points ofintersection of lines connecting opposing ends of adjacentrepresentative lines, and estimating a substantially vertical linethrough the point of intersection, wherein the difference between thesubstantially vertical line and a line perpendicular to the aggregatedslope indicates the accuracy of the refining angle of rotation; androtating the first rotated image by the refining angle of rotation toresult in a final rotated image.
 14. The computer program product ofclaim 13, wherein said determining the initial angle of rotation,includes: sampling a plurality of test angles at regular angle intervalsof rotation.
 15. The computer program product of claim 13, wherein saiddetermining the initial angle of rotation, includes: sampling aplurality of test angles in a range of −90 degrees to +90 degreesrotation.
 16. The computer program product of claim 13, wherein saiddetermining the initial angle of rotation, includes for each resultantrotated image of a sample test angle: converting the resultant rotatedimage into a pixel map with each pixel in the pixel map being determinedto be empty or non-empty by applying a pixel content threshold;weighting each line of pixels in the pixel map according to a number ofnon-empty pixels in said each line; determining an empty line in thepixel map as a line of pixels in the pixel map with a weight less than aline content threshold; and counting a number of empty lines in theresultant rotated image.
 17. The computer program product of claim 13,said method further comprising: determining the text line areas of thefirst rotated image by forming areas of adjacent pixel lines, whereinthe adjacent pixel lines are each above a threshold weight of non-emptypixels, and wherein the determined text line areas are separated by atleast one pixel line below the threshold weight of non-empty pixels.